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(54) Manipulator welding apparatus 
with vision correction for 
workpiece sensing 



(57) During a teach mode, the 
manipulator is taught the desired 
welding path on a workpiece by 
recording data representing the 
welding path as an operator controls 
manipulator. Data representing a 
reference image or template provided 
by a vision system including a camera 
carried by the manipulator arm is also 
recorded. As successive workpieces are 
presented to the manipulator for 
performing the desired welding path, in 
a repeat work cycle mode the 
manipulator is controlled to move in 
accordance with the recorded taught 



data in a first repeat pass wherein 
images of the welding seam region are 
detected at successive points along the 
taught path. The vision system, utilizing 
an image processor, detects the 
deviation between the taught welding 
path and the actual welding path of the 
workpiece. In response to the deviation 
data provided by the image processor 
in the first repeat pass corrects the 
recorded taught data to provide 
corrected welding path data for the 
particular workpiece orientation and 
geometry encountered by the 
manipulator. During a second repeat 
pass of the repeat work cycle mode, 
welding of the actual workpiece seam is 
performed in accordance with the 
corrected welding path data obtained in 
the firsj repeat pass. 
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SPECIFICATION 

Method and apparatus for manipulator welding 
apparatus with vision correction for workpiece sens- 
5 ing 

The present invention relates generally to man- 
ipulator welding apparatus and more particularly to 
control apparatus utilizing a vision correction system 
1 0 for workpiece sensing to operate manipulator weld- 
i ing apparatus to visually detect the deviation bet- 
ween a taught welding path and the actual seam to 
be welded during a first step and to perform welding 
.along the actual seam of a particular workpiece dur- 
1 5 ing a second step. 

Various welding apparatus and vision detection 
systems have been proposed. For example, U.S. 
application Serial No. 106,039 filed by I. Masaki on 
December 21, 1979 provides a method and 
20 apparatus for detecting any positional deviation of a 
workpiece from a reference position by means of a 
visual detection system and an image processor. 
Further, U.S. application Serial No. 066,323 filed by I. 
Masaki on August 14, 1979 and U.S. application Ser- 
25 ial No. 077,320 filed by I. Masaki on September 20, 
1979 disclose arrangements for interrupting the 
welding operation and performing the detection of 
images of the workpiece during the time that the 
welding is interrupted or the welding current is 
30 decreased. Further welding apparatus with control 
arrangements are disclosed in U.S. patent No. 
4,1 15,684 which issued to T. H. Lindbom on Sep- 
tember 19, 1978, U.S. patent No. 4,086,522 which 
issued to J. F. Engelberger et al on April 25, 1978, 
35 U.S. patent No. 4,030,617 which issued to H. Richter 
on June 21, 1977, U.S. patent No. 4,105,937 which 
issued to G. Tuda et al on August 8, 1978 and U.S. 
application Serial No. 154,439 filed by W. Perzley et 
alon May 29, 1980. 
40 While the above described arrangements are gen- 
erally suitable for their intended use, it is difficult to 
obtain image data of the sensed workpiece during 
welding. Even in the arrangements where provisions 
are made to decrease welding current or interrupt 
45 welding to obtain images of the workpiece, optical 
noise in the vision system is still a problem. Further, 
the vision system is subjected to the welding envi- 
ronment with possible detrimental effects on the vis- 
i ion system. Additionally, if the sensing of the work- 
50 piece welding seam is performed during the actual 
welding of the seam, the vision system carried on 
. the manipulator arm must be positioned near the 
weld tip. In addition to the vision system being 
exposed to the welding environment and optical 
55 noise resulting therefrom, the weld gun and vision 
system arrangement may cause clearance problems 
with obstructions on or around the workpiece and 
may present various obstacles to the manipulator 
hand being controlled to move overthe welding 
60 path. Further, if image data of the workpiece is 
obtained during the welding operation to provide 
path correction data, the path computation data 



must be calculated during the welding process and 
as the manipulator arm moves overthe welding 
65 path. 

Accordingly it is a principal object of the present 
invention to provide a control method and apparatus 
for manipulator welding apparatus having a vision 
correction system for workpiece sensing that over- 
70 comes one or more of the above described disad- 
vantages of the prior art arrangements. 

It is another object of the present invention to pro- 
vide a control method and apparatus for man- 
ipulator welding apparatus having a vision correc- 
75 tion system for workpiece sensing wherein a desired 
welding path is taught on a workpiece during a teach 
mode, and for each successive workpiece presented 
to the manipulator a vision correction system 
detects the deviation between the taught path and 
80 the actual workpiece path to be welded during a first 
repeat pass overthe workpiece, corrected path data 
is calculated from the deviation data that represents 
the desired welding path for the particular workpiece 
position and the corrected path data is utilized dur- 
85 m 9 3 second repeat pass to perform the welding 
along the desired path. 

It is another object of the present invention to pro- 
vide method and apparatus for manipulator welding 
apparatus wherein path correction data is obtained 
90 for a particular workpiece representing the deviation 
between a taught path and an actual workpiece path 
accounting for changes in workpiece position before 
the manipulator performs the welding of the desired 
welding path. 
95 It is a further object of the present invention to 
provide a manipulator hand for a manipulator arm 
including a vision system and a weld gun wherein 
the manipulator hand may be rotated to selectively 
position either the vision system or the weld gun at a 
100 work location. 

It is another object of the present invention to pro- 
vide a method and apparatus for manipulator weld- 
ing apparatus with a vision correction system for 
performing corrections to a taught work path 
105 wherein the detection of the actual work path and 
the correction of taught work path data is accomp- 
lished independently of the welding of the work- 
piece. 

It is yet another object of the present invention to 
110 provide a method and apparatus for manipulator 
welding apparatus utilizing a vision correction sys- 
tem that corrects a taught path for changes in work- 
piece position frorn a taught workpiece location dur- 
ing a first repeat pass overthe workpiece at a reia- 
115 tively high speed compared to that of the welding 
speed and prior to the time of a second repeat pass 
during which the welding is accomplished in accor- 
dance with the corrected path data. 

Briefly, these and other objects of the present 
120 invention are achieved by providing control 

apparatus for manipulator welding apparatus for 
accurately performing a taught welding path on suc- 
cessive workpieces and accounting for deviations in 
workpiece position from the workpiece on which the 



The drawings originally filed were informal and the print here reproduced is taken 

from a later filed formal copy. 
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weld path was initially taught. The control apparatus 
includes a vision correction system for workpiece 
sensing. During an initial teach mode, the man- 
ipulator is taught the desired welding path on a 
5 workpiece by the appropriate recording of data rep- 
resenting the welding path as an operator controls 
movement of the manipulator. In addition to the data 
representing the taught welding path, data repres- 
enting a reference image or template is also 

10 recorded in the teach mode. The reference image or 
template data is provided by a vision system includ- 
ing a camera carried by the manipulator arm. As 
successive workpieces are presented to the man- 
ipulator for performing the desired welding path, in 

15 a repeat work cycle mode the manipulator is control- 
led to move in accordance with the recorded taught 
data in a first repeat pass wherein images of the 
welding seam region are detected at successive 
points along the taught path. The vision system 

20 utilizing an image processor visually detects the 
deviation between the taught welding path and the 
actual welding path of the workpiece. The image 
processor by means of the visual detection in the 
first repeat pass provides deviation data represent- 
*25 «"9 tne deviation between a taught standard path 
and the actual welding path of the workpiece. The 
control apparatus in response to the deviation data 
provided by the image processor in the first repeat 
pass corrects the recorded taught data to provide 

30 corrected welding path data for the particular work- 
piece position encountered by the manipulator. Dur- 
ing a second repeat pass of the repeat work cycle 
mode, welding of the actual workpiece seam is per- 
formed in accordance with the corrected welding 

35 path data obtained in the first repeat pass. When 
successive workpieces are provided to the man- 
ipulator, the first and second repeat passes of the 
repeat mode are performed for each workpiece. Dur- 
ing the teach mode, the manipulator is taught the 

40 desired welding path with a weld tip in the appropri- 
ate welding position. Further a camera of the vision 
system carried by the manipulator arm during the 
teach mode is appropriately positioned to provide 
the reference image or template. During the first 

45 repeat pass of the repeat mode, the vision system on 
the manipulator arm is appropriately positioned to 
move over the taught path. In the second repeat pass 
of the repeat mode, the weld gun is positioned to 
perform the welding. Any positional offsets between 

50 the optical center of the vision system on the man- 
ipulator arm and the weld point of the weld gun are 
accounted for by the control apparatus to provide 
appropriate data. In one arrangement, the vision sys- 
tem and the weld gun of the manipulator arm are 

55 provided by portions of the vision system being 
mounted on the manipulator arm along with the 
weld gun. In a first specific embodiment of this 
arrangement, the vision system and the weld gun 
are presented to the workpiece in an aligned fashion. 

60 In a second specific embodiment of this arrange- 
ment, the vision system and the weld gun are 
mounted on the hand with appropriate rotation of 
the manipulator hand at the end of the robot arm 
selectively presenting either the vision system or the 

65 weld gun. In another arrangement, two separate 



manipulator hands are provided; one hand carrying 
the appropriate portions of the vision system such as 
a camera head and an optical pattern projection unit 
that is utilized for detection and a second hand 

70 including the weld gun to perform the welding and 
to teach the desired welding path. 

The invention both as to its organization and 
method of operation together with further objects 
and advantages thereof will best be understood by 

75 reference to the following specification taken in con- 
junction with the accompanying drawings. , 
FIG. 1 is a perspective view of a programmable man- 
ipulator adjacent a workpiece welding station and a 
functional representation of the control apparatus of 

80 the present invention; 

FIG. 2 is a perspective view of another form of a 
programmable manipulator adjacent a workpiece 
welding station and a block diagram representation 
of the control apparatus of the present invention of 

85 

FIGS. 3 and 4 are perspective views representative 
of respective workpieces to be welded in accordance 
with the control apparatus of FIGS. 1 and 2 and illus- 
trating projected light patterns of the vision system 
90 of the present invention; 

FIG. 5A is a graphical plan view representation and 
FIG. 5B is a front elevational view of a taught weld 
path on a workpiece and a corrected weld path in 
accordance with the principles of the present inven- 
95 tion; 

FIG. 6 is a flow diagram illustrating the method of 
the present invention and the general operation of 
the control apparatus of the present invention of 
FIGS. 1 through 5 to accomplish welding of a work- 
100 piece; 

FIG. 7 is an enlarged, elevational view of the man- 
ipulator arm of the present invention of FIGS. 1 and 2 
and illustrating one hand arrangementto practice 
the present invention including a visual detection 
105 system and a weld gun. 

FIG. 8 is an elevational view of a weld gun hand; 

FIG. 9 is an elevational view of a vision system 
hand that provides minimal interference to the work 
piece environment; and 
110 1 0 is an elevational view of an alternate vision 

system hand. 

Referring now to FIG. 1, manipulator apparatus or 
robot 10 is illustrated adjacent a work station 1 2 to 
accomplish the welding of a workpiece generally 
115 referred to at 14 in accordance with the principles of 
the present invention. The workpiece 14, for exam- 
ple, as illustrated in FIG. 1 includes two metal plates 
16, 18 which are to be welded along the abutting 
seam formed therebetween. The manipulator 10 
120 includes a manipulator arm 20 that is provided with 
a hand or tool receiving portion 22. A manipulator 
hand arrangement 24 is positioned on the receiving 
portion 22. In one arrangement of the present inven- 
tion, the hand 24 carries a weld gun 26 and a camera 
125 or other suitable imaging device 28. An optical pat- 
tern projection unit 30 is arranged either on the hand 
24 or at a suitable fixed reference position indepen- 
dent of the hand. The weld gun 26 is connected to 
welding equipment referred to generally at 32 that 
130 provides a welding wire feed system at 34 to the 
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weld gun 26 along with other suitable electrical 
interconnections and a welding gas supply in accor- 
dance with conventional welding practices. 
The camera 28 is controlled by and provides imag- 
5 ing signals to a camera controller arrangement 36 
over signal and control lines 38. The camera control- 
ler arrangement 36 is connected to provide signals 
to an image processing unit 40. The camera 28, the 
optical projection unit 30, the camera controller 36 
1 o and the image processing unit 40 comprise a vision 
. system substantially as shown and described in the 
aforementioned copending Masaki application Ser- 
ial Nos. 077,320, 066,323 and 106,039. 
The image processing unit 40 is interconnected 
1 5 over data lines 42 to a data transformation system 
and system controller 44 of the present invention. 
The data transformation system and system control- 
ler 44 controls operation of the manipulator welding 
system and controls operation of the manipulator 10 
20 bv means of a servo control system referred to gen- 
erally at 46. The data transformation system and sys- 
tem controller 44 supplies the appropriate control 
signals to the servo control system 46 as command 
signals to position the manipulator arm 20 in one or 
25 more controllable axes. 

In accordance with important aspects of the pres- 
ent invention and in one specific embodiment in an 
initial teach mode an operator controls the man- 
ipulator arm 20 in the controllable axes to move the 
30 tip of the weld gun 26 over a desired welding path 
with respect to the taught workpiece 14. During the 
teach mode, appropriate welding path data is 
recorded in the system controller 44 representing 
taught points along the desired welding path. 
35 Further, during a predetermined portion of the teach- 
ing mode, the projection unit 30 is operatively 
positioned to project a light pattern on the workpiece 
14. In response to the projected light pattern, the 
camera 28 along with the camera controller 36 and 
40 the image processing unit 40 provides a reference or 
template image for storage along with the taught 
path data. The projection unit 30 projects an optical 
slit pattern on the workpiece 14 and the vision sys- 
tem components including the camera 28, the cam- 
45 era controller 36 and the image processing unit 40 
detect the optical pattern as transformed by the 
shape of the surface of the workpiece 14. The refer- 
ence or template image is taken at one or more suit- 
able locations along the taught welding path as 
* 50 dependent upon the geometry of the welding path. 
During the teach mode, welding data is also 
recorded including desired weld speed and other 
appropriate parameters of the welding operation. In 
another specific embodiment, only the reference 
55 template image is taken with a reference work piece 
and the taught welding path is recorded during the 
first repeat pass of the first repeat workpiece as well 
be explained in detail hereinafter. The taught path 
for the first repeat work piece is then utilized for suc- 
60 cessive work pieces. 

In accordance with important aspects of the pres- 
ent invention, as successive workpleces are pre- 
sented to the manipulator 10 for welding, the man- 
ipulator 10 is controlled to perform a repeat mode 
65 including a first repeat pass during which the man- 



ipulator arm 20 is controlled to move over the taught 
welding path at a speed much higher than weld 
speed in accordance with the stored taught data with 
the camera 28 operatively positioned over the taught 
70 welding path. 

During the movement of the manipulator arm 20 
in the first repeat pass over the taught welding path, 
the image processing unit 40 over data lines 42 pro- 
vides deviation data from the detected workpiece 
75 images at suitable points along the welding path 
representing the deviation of the actual welding 
seam presented by the workpiece 14 from the taught 
welding path template image. In one specific embod- 
iment, the system controller 44 in accordance with 
80 the stored taught data and the deviation data from 
the image processing unit 40 calculates and stores 
corrected welding path data for the present work- 
piece 14 accounting for any changes in location of 
the workpiece 14 from that of the taught reference 
85 workpiece in the teach phase. 

In a second repeat pass of the repeat mode, the 
weld gun 26 is operatively positioned over the work- 
piece 14 and the corrected path data calculated and 
stored in the first repeat pass is utilized to control the 
90 manipulator arm 20 to weld the desired welding path 
on the workpiece 14 in cooperation with the opera- 
tion of the welding equipment 32. 

Thus, the desired welding path is taught and 
recorded on a reference workpiece 14 during a single 
95 teach mode and as successive workpieces 14 are 
presented to the manipulator 10, the repeat mode 
with the two repeat passes is performed for each of 
the workpieces. 
Referring now to FIG. 2, another type of man- 
100 ipulator 10 is illustrated for operation on a workpiece 
14. The system controller 44 of the present invention 
is shown with appropriate data and control inter- 
connections with the servo control system of the 
robot controller 46, the image processor 40, the 
105 camera 36, the welding equipment 32 including a 
welding controller 50 and a welding power source 
52, and a laser driver stage 54 for operation of the 
projection unit 30 as a laser head. 
The manipulator 10 suitable for use in practice of 
110 the present invention includes various types of prog- 
rammable manipulators or robots. For example, a 
manipulator 10 as shown in FIG. 1 is commercially 
available from Unimation, Inc. as a UNIMATE 
(Trademark of Unimation, inc.) type 2000 man- 
115 ipulator and the manipulator 10 of FIG. 2 is commer- 
cially available from Unimation, Inc. as a PUMA 
(Trademark of Unimation, Inc.) type 500 arm. The 
PUMA type 500 manipulator arm shown in FIG. 2 
includes a microprocessor based control system 
120 operating in accordance with VAL (T rademark of 
Unimation. Inc.) programming and control system 
for computer controlled robots and manipulators. 
Reference may be made to the publication "User's 
Guide to VAL, A Robot Programming And Control 
125 System" version 11, February, 1979, 2nd Edition for 
a more detailed discussion of the operation of this 
type of manipulator. 

The system controller 44 of FIGS. 1 and 2 in one 
arrangement of the present invention is operable as 
130 a VAL robot programming and control system as 
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additionally enhanced by the data transformation 
system of the system controller 44 as will be 
explained in more detail hereinafter. 
The system controller 44 for the manipulator 10 of 
5 FIG. 1 in other arrangements utilizes the respective 
control system as further described in U.S. patent 
Nos. 3,661,051, 4,086,522, 4,163,183 and U.S. appli- 
cation Serial No. 1 54,439 filed by W. Perzley et al on 
May 29, 1980 to which reference may be made for a 
10 more detailed discussion of suitable types of control 
system for use in connection with the present inven- 
tion. 

Further, the operation of the vision system of 
FIGS. 1 and 2 including the camera 28, the projection 

15 unit 30, the camera controller 36 and the image pro- 
cessing unit 40 for providing a deviation output 
signal by comparison of a reference tern pi ate image 
and an actual image is disclosed in the aforemen- 
tioned copending Masaki application Serial Nos. 

20 077,320,066,323 and 1 06,039 to which reference 
may be made for a more detailed discussion of the 
structure and operation of the vision system of FIGS. 
1 and 2. 

Considering now a brief discussion of typical 

25 workpieces 14 to illustrate the operation of the pres- 
ent invention and referring now to FIG. 3, the projec- 
tion unit 30 projects an optical slit pattern onto the 
generally perpendicularly arranged workpiece por- 
tions 60, 62. The workpiece portions 60, 62 transform 

30 the optical pattern projected by the unit 30 in accor- 
dance with the surface shape and position of the 
workpiece portions; for example, as depicted by line 
pattern 64 on workpiece portion 62 and line pattern 
66 on workpiece portion 60. This type of optical pat- 

35 tern projection is useful for path correction systems 
including arc welding, sealing, brazing and the like. 

Thus, the camera 28 detects the transformed light 
pattern image including image pattern portions 64 
and 66; the intersection of the lines 64, 66 defining 

40 the points along the intersection seam 68 of the 
workpiece portions 60, 62 to be welded for various 
positions of the camera 28 and the projection unit 30. 

Thus, considering both the reference workpiece in 
the teach mode and the successive workpieces 

45 encountered during the various repeat modes there- 
after, the camera 28 detects the image portions 64, 
66 of the actual welding seam 68 in accordance with 
the position of the workpiece portions 60, 62. 
Referring additionally to FIG. 4, a second example 

50 of a welding application is illustrated therein for 
welding the lap seam between the generally planar 
and overlapping workpiece portions 70, 72. The opti- 
cal projected slit pattern from the projection unit 30 
forms an image as transformed by the shape of the 

.55 workpiece portions 70, 72 including a first line image 
74 on the workpiece portion 70 and a second line 
portion 76 on the workpiece portion 72. 

Thus the respective workpiece images of FIGS. 3 
and 4 are typical of the reference template image 

60 recorded by the image processor 40 in the teach 
mode for a reference workpiece and are also typical 
of the actual images formed by the successive work- 
pieces in the repeat modes as detected by the cam- 
era 28 during the first pass of the repeat mode in 

65 accordance with operation on each successive 



workpiece. 

Considering the taught welding path obtained in 
the teach mode and the corrected welding path 
obtained in the first repeat mode and referring to 

70 FIG. 5, an illustrative example of a taught path 68 
includes the taught points A, B, C and D for example 
defining the straight line welding seam 68 to be 
welded on the workpiece portion 60, 62 of FIG. 3. 
Thus during the teach mode, data representing the 

75 taught points A, B, C and D are recorded and stored 
in the system controller 44 in accordance with 
movement of the weld gun 26 over the taught path 
68. Additionally, the reference image or template 64, 
66 detected by the camera 28 is also stored in the 

80 image processor 40. 

In one specific arrangement of the present inven- 
tion, the reference image 64, 66 is recorded at only 
one point along the teach path 68 since the geometry 
of the weld seam 68 results in a constant image 

85 being obtained at the various points along the 
desired welding path. 

In the first repeat pass of the repeat mode for a 
workpiece 14 presented to the manipulator 10 and 
with the camera 28 in the operative position, the 

90 manipulator 10 is controlled to move in accordance 
with the taught path data at a relatively high rate of 
speed with the camera 28 obtaining slit pattern 
images resulting from the projected pattern from 
unit 30 on the workpiece 14 at the various taught 

95 points A, B, C and D in one specific arrangement. 
Thus at the taught points A, B, C and D, the image 
processing unit 40 calculates and provides respec- 
tive deviation data D a , D b , D c and D d representing 
two-dimensional deviation data in an X-Z reference 

100 plane and including AX and AZ components. In one 
specific arrangement, the system controller 44 dur- 
ing the first repeat pass and in accordance with the 
deviation data Da, Db, D c and D d and the taught data 
representing the points A, B, C and D, calculates a 

105 corrected welding path represented by the corrected 
data points, A', B', C' f and D' defining the actual 
welding seam presented by the newly positioned 
workpiece 14 representing positional changes from 
the taught reference workpiece position in the X and 

1 10 Z reference axes. In another specific arrangement, 
the system controller 44 utilizes absolute position 
data provided by digital encoders of the manipulator 
apparatus 10 and the deviation data Da, D c and D d 
to provide the corrected data points A', B', C, and D'. 

115 Refering now specifically to FIG. 5 A, the plan view' 
of the taught path 68 and the corrected path 80 rep- 
resents for example the plan view of the workpiece 
weld path of FIG. 3. In FIG 5A, the AX components of 
the deviation D in depicted. In Fig. 5B, the AY devia- 

120 tion component illustrates the deviation in the Y-Z 
reference plane an detected by the image processing 
unit 40. The AZ deviation components are calculated 
by the image processing unit 40 in accordance with 
the relationships AZ = AY tan 0, where 0 is the 

125 depicted angle in FIG. 5B formed between the X-Y 
reference plane and the projection unit 30. 

In the specific arrangement where the present pos- 
ition of the manipulator is utilized in conjunction 
with the deviation data at the respective point to 

130 provide the corrected path data 80, inaccuracies due 
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to servo loop positioning in response to the taught 
data do not affect the actual corrected data for the 
welding path. Further such an arrangement is suit- 
able wherein in the first repeat pass it is desirable to 
5 take the images at points other than the taught 
points for various reasons. Thus the present position 
of the manipulator is utilized in accordance with the 
deviation data at the respective present positions to 
obtain the corrected welding path data. 
10 Considering nowthe basic operation of the system 
. controller 44 and in particular the data transforma- 
tion system provided therein and referring now to 
FIG. 6, the flow diagram of FIG. 6 represents the 
- operation of the present invention including the 
15 additional functions performed by the system con- 
troller 44 in addition to available manipulator control 
arrangements such as the PUMA type 500 arm with 
VAL control programming discussed hereinbefore. 
The start of the flow program of the system control- 
20 ler 44 proceeds through the teach mode 84 wherein 
the function block 86 represents the teaching of the 
reference template image in accordance with the 
data from a vision system with the camera 28 in the 
operative position and the projection unit 30 project- 
25 ing the slit pattern resulting in image 64, 66 of FIG. 3 
onto the taught reference workpiece. 

After teaching the template image the program 
flow proceeds to a function block 88 representing the 
teaching of the desired welding path on the refer- 
30 ence workpiece 14 for example by the movement of 
the manipulator along the welding path 68 of FIG. 3 
and the storing of data representing the taught 
points A, B, C and D of FIG. 5. The program flow 
proceeds to a function block 90 wherein other 
35 appropriate teaching information is recorded such 
as the desired welding speed and other parameters 
of the welding operation. The teach step 90 also 
includes appropriate weld gun and camera position 
definition data as may be necessary due to offsets of 
40 the weld gun point and optical center point of the 
manipulator hand from a reference point. In accor- 
dance with VAL programming, the teach step SO 
includes the inputting of data representing and 
defining either the offsets of the hand 24 in the cam- 
45 era and weld gun positions or two separately 

attached hands in a specific arrangement wherein a 
camera hand and a separate weld gun hand are pro- 
vided. 

The program flow of the system controller 44 then 
50 proceeds to the repeat modes 91 and specifically at 
flow point 92 into the first repeat mode or pass 94, In 
t the first repeat mode 94, a function block 96 repres- 
ents operation of the servo controller 46 moving the 
manipulator 10 at a high rate of speed to the first 
55 taught point. After the manipulator has been moved 
to the first taught point with the camera 28 in the 
operative position over the workpiece 14, an actual 
image at taught point A is obtained from the vision 
system. The flow of the program proceeds to a func- 
60 tion block 100 wherein the image processing unit 40 
calculates the deviation D a and provides this devia- 
tion to the system controller 44. 

After the image processing unit has provided the 
deviation data D a representing the deviation bet- 
65 ween the taught point A and the actual weld seam 



point represented by the workpiece, the program 
flow proceeds to a function block 1 02 wherein the 
system controller44 calculates the position of the 
actual seam as data point A' from the deviation data 
70 Da and the taught data point A. When the calculation 
of the actual point A' is completed, the program flow 
proceeds to the decision block 104 to determine 
whether or not the first repeat pass has been com- 
pleted encompassing the calculation of corrected 
75 path data for each of the taught data points, for 
example A, B, C and D. In the present example, 
whereupon particular data point A' has just been 
calculated, the result in decision block 104 is NO and 
the program flow proceeds via signal line 106 back 
80 to program flow point 92 to the function block 96. In 
function block 96, the taught data is incremented by 
one data point and the first repeat pass flow 94 con- 
tinues to calculate the corrected data B', C and D' 
corresponding to the respective taught points B, C 
85 and D when the manipulator arm 20 is moved to 
each of the taught points B, C and D. It should be 
understood however that the flow diagram of FIG. 6 
is merely illustrative of one specific embodiment of 
the present invention. For example in another 
90 specific embodiment the corrected data A', B', C and 
D' are calculated after the arm has been moved to all 
the taught points A, B, C and D and the respective 
deviation data D a , D b , D c and D d are obtained. £ 
When the last corrected path point such as D' has a 
95 been calculated in accordance with the image taken 
from the taught point D, the result in decision block 
104 is yes and the program flow proceeds through 
flow point 108 to the second repeat mode or second 
repeat pass 11 0 of the repeat mode 91 . In the second 
100 repeat mode 110, a function block 112 proceeds to 
condition the manipulator 10 for welding the actual 
seam of the workpiece in accordance with the actual 
welding path 80 defined by the corrected data points 
A', B' C and D' stored in the system controller 44 as a 
105 result ofthe first repeat mode 94. Now with the weld 
gun 26 in the operative position, the function block 
112 proceeds to condition the manipulator 10 to 
weld the actual seam with appropriate control ofthe 
welding equipment 32 and the desired recorded 
1 10 speed of movement along the actual path 80 in 

accordance with the data entered in the teach mode. 
Thus, the manipulator arm 20 is controlled to move 
the weld gun 26 over the path defined by the points 
A', B', C'andD'. 
115 After the completion ofthe weld path 80, the prog- 
ram flow proceeds to the decision block 1 14 to 
determine whether the next workpiece to be welded 
is in position. If the determination is YES signifying 
that another workpiece Is to be welded and is in posi- 
1 20 tion for welding, the program flow proceeds from 
the decision block 114 over the signal path 116to the 
flow point 92 at the beginning ofthe first repeat 
mode 94. 

Thus for the next workpiece, the first and second 
125 repeat modes 94 and 110 respectively are accomp- 
lished with the next workpiece being sensed in the 
first repeat mode, the actual welding path being cal- 
culated, and the welding path being welded in the 
second repeat mode 110. 
130 If the decision, in the block 1 14 results in an indica- 
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tion that there is no additional worlcpiece to be 
welded, the program flow proceeds to an end of 
program function 118 with appropriate termination 
of the program. The next workpiece indication in the 
5 block 1 14 is provided In specific arrangements by 

^ either external operator input, or by control inputs of 
the system controller 44 via automated workpiece 
transfer apparatus, or by sensing function per- 
formed by the manipulator 10. 

10 Considering now the details of the manipulator 
hand 24 that selectively positions either the weld 
gun 26 or the camera 28 and projection unit 30 of the 
vision system to the operative position over the 
welding path of the workpiece and referring now to 

15 FIG. 7, the weld gun 26 is positioned and carried on 
the manipulator hand 24 at a position approximately 
180 degrees apart from the optical pattern projection 
unit 30 and the camera 28. Thus, the weld gun 26 
defines a weld point 122 and the camera 28 and opti- 

20 cal pattern projection unit 30 define an optical center 
reference point 124 at the intersection of their 
respective optical axes. The weld point 122 and the 
optical center point 124 are approximately 180 
degrees apart on the hand 24. The manipulator arm 

25 20 includes a controllable axis defining an articu- 
lated joint at 126 providing a yaw or wrist swivel 
movement to rotate the hand 24 to provide the pre- 
sentation of either the weld gun 26 at the weld point 
122orthe vision system at optical center point 124 

30 over the welding seam of the workpiece. 

Thus with appropriate programming of the system 
controller 44 or by the input of a teach operator on 
the manipulator controls, the manipulator arm 20 
selectively presents either the optical center point 

35 1 24 or the weld point 1 22 to the workpiece. As dis- 
cussed hereinbefore, the weld point 122 is presented 
during portions of the teach mode and during the 
second repeat pass. Further, the optical center point 
is positioned over the workpiece during a portion of 

40 the teach mode and also during the first repeat pass. 
In the situation where the selective rotation of the 
hand 24 through wrist swivel movement at 126 
results in different offsets between the optical center 
point 1 24 and the weld point 1 22 with respect to the 

45 receiving portion 22 of the manipulator arm, the sys- 
tem controller 44 includes appropriate hand defini- 
tion data including the transformation of coordinates 
defining the various respective offset of the optical 
center point and the weld point to appropriately pos- 

50 ition either the optical center point or the weld point 
at the stored data points representing either taught 
data or corrected path data. Further, during the teach 
mode, the offset data of the weld point and the opti- 
cal center point are utilized to allow accurate posi- 

55 tioning of the camera 28 or the weld gun 26 and the 
appropriate recording of position data defining man- 
ipulator arm position in each of the controlled axes. 

In another specific embodiment, the weld gun 26, 
and the camera 28 are mounted on the hand 24 such 

60 that both the camera axis and the weld point 122 are 
simultaneously aligned with and presented to the 
workpiece. Further, the projection unit 30 is also 
mounted on the hand 24 with the optical center point 
124 being aligned with the work piece simultane- 
65 ously with the weld point 122. 



Considering now FIG. 8, 9 and 10 and the specific 
arrangement where a separate weld gun hand 140 
(FIG. 8) and a separate camera and projection unit 
hand 142 (FIG. 9) or 144 (FIG. 10) are provided, the 
70 manipulator automatically and selectively attaches 
one of the hands as required for the particular prog- 
rammed operation as explained in connection with 
FIG. 6. The exchange between the hands is also 
automatically provided by the manipulator 10 in 
75 accordance with the programmed operation of the 
system controller 44. The hand not currently being . 
used by the manipulator 10 is placed in a hand hol- 
der (not shown). The manipulator 10 in accordance 
with programmed control automatically exchanges 
80 the hands from the respective hand holder stations 
as required by placing one hand in one hand holder 
station and picking up the second hand in a second 
hand holder station. Of course, the various trans- 
formation offsets between each of the hands and the 
85 manipulator arm hand reference point at the hand 
attachment are also stored by the system controller 
to enable proper positioning of each of the hands 
and the accurate recording of arm position data. It 
should be noted that the various embodiments of 
90 the hands 24, 140, 142 and 144 provide minimal 

intrusion problems to the v/ork piece environment to 
avoid obstructions in the work piece environment. 

Considering now the operation of the present 
invention concerning the data transformation sys- 
95 tern of the system controller 44, a generalized coor- 
dinate transformation analysis will be discussed first 
that applies to generalized manipulator operations. 

The deviation data obtained from the image pro- 
cessing unit 40 in the first repeat mode is presented 
100 to the system controller 44 in coordinates relative to 
the orientation of the coordinate axes of the camera 
28. The deviation data at a particular point i referred 
to hereinbefore as Du for example, is represented by 
X and Z components AXi and AZ U respectively denot- 
105 «ng horizontal and vertical deviation as referenced in 
FIG. 5. The deviation in a matrix format is defined as 
follows: 
[D] = 



no 



I" 1 0 0 AX, "I 

10 10 0 j 

0 0 1 AZ, 1 

L 0 0 0 1 -J 



Appropriate reading of the deviation components is 
provided by the system controller 44. 

A transformation matrix [C w ] from camera coordi- 
nates to a world generalized coordinate reference of 
1 15 the manipulator is obtained as follows: 
[Cj =[Rw] x[Cj, 
where [R J represents the transformation matrix 
from wrist reference coordinates of the manipulator 
to generalized world coordinates, and [dj repres- 
120 ents the transformation matrix from camera refer- 
ence coordinates to wrist coordinates. 

Thus, a target or destination point [P w ] in the 
repeat mode for welding includes the corrected 
coordinates for the weld gun in generalized world 
125 coordinates and is defined as follows: 
[Pw]=[Cjx[D]. 

To properly control the position of the manipulator 
arm, it is necessary to define [R' w ] the coordinate 
matrix that describes the corrected position and 
130 orientation for the manipulator wrist in generalized 
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world coordinates. The matrix [R' w ] is obtained as 
follows: 

[R'w]=[Pw]x[T r ]-\ 
where the matrix [TV]" 1 represents the transforma- 
5 tion matrix from the weld gun coordinates at the. 
weld point 122 to the wrist reference coordinates. 

Thus, the matrix [R' w ] is used to control operation 
and positioning of the manipulator 10 in the second 
repeat mode in accordance with the following rela- 
10 tionship: 

. [R'wJ=[C w ]x[D]x[T r ]^. 

Considering now s more detailed and specific 
application of the present invention utilizing a PUMA 
. type 500 manipulator having a VAL programming 
1 5 and control system, the taught point data is utilized 
directly to obtain the corrected weld path coordi- 
nates in accordance with the deviation data obtained 
in the first repeat pass. 
Specifically, the deviation obtained at a particular 
20 taught point i is represented as[DEV]i and repres- 
ents the deviation defining the corrected weld point 
relative to the camera coordinates in the first repeat 
pass. The corrected weld point [REPEAT],- is obtained 
as follows: 
25 [REPEAT] i = [TAUGHT] i x [DEV]i 

resulting in the matrix definition of the corrected 
point A' corresponding to the taught point A for 
example. 

The specific variations and additions to the VAL 
30 program and control system to practice the present 
invention includes the definition of a MOVCAM A; A' 
function or instruction that defines the movement of 
the manipulator arm to location A, reading the cam- 
era deviation [DEV] at location A via the image pro- 
•35 cessor, and defining the location of a corrected point 
A' therefrom. 

Considering the specific program steps to be 
accomplished to practice the present invention, the 
following program steps are performed as organized 
40 into PROGRAM TEACH and PROGRAM REPEAT 
segments as follows and corresponding to the flow 
diagram of FIG. 6 (illustrating a taught path of 4 
points): 

PROGRAM TEACH 
45 1. TOOL WELD 

2. SPEED WELD 

3. WELD ON 

4. MOVE A 

5. MOVE B . 
50 6. MOVE C 

7. MOVE D 

8. WELD OFF 

PROGRAM REPEAT 
REPEAT PASS ONE 
55 1. TOOL CAMERA 

2. SPEED REPEAT 

3. MOVCAM A, A' 

4. MOVCAM B, B' 

5. MOVCAM C, C 
60 6. MOVCAM D,D' 

REPEAT PASS TWO 

7. TOOL WELD 

8. SPEED WELD 

9. WELD ON 
65 10. MOVE A' 



11. MOVE B' 

12. MOVEC 

13. MOVED' 

14. WELD OFF 

70 Additionally, during the PROGRAM TEACH phase, 
an operator teaches a reference image template as 
described hereinbefore. Further, the two tools CAM- 
ERA and WELD GUN are defined corresponding to 
the respective TOOL CAMERA and TOOL WELD 
75 steps with the VAL program and control system 
incorporating the TOOL[<transformation>] prog- 
ram instruction setting the value of tool transforma- 
tion to the value defined in the brackets transfor- 
mation^ The effect of the TOOL command results in 
80 the monitoring by the VAL system to account for the 
tranformation offset matrix of the defined tool; i.e., 
an internal matrix representing the tool offset coor- 
dinates is automatically taken into consideration 
each time data is to be recorded or each time the 
85 manipulator is moved to a defined point. 

The SPEED WELD of step 2 in PROGRAM TEACH 
enters the desired welding speed, for example 5 
units/second. The SPEED REPEAT of step 2 of 
REPEAT PASS ONE enters the desired repeat speed, 
90 for example, 200 units/second. 

Concerning the preferred embodiment of the pre- 
sent invention, the PROGRAM TEACH sequence is 
not performed and only the PROGRAM REPEAT 
sequence of steps 1 -1 4 is necessary along with the 
95 teaching of a reference image template. Then, in the 
preferred embodiment and during the repeat opera- 
tion on the first repeat workpiece, the MOVCAM 
steps include movement by the operator and the 
teach controls to the designated points A, B, C and D 
100 ' n addition to the generation of the deviation via the 
image processor and the defining of the corrected 
data points A', B', C and D'. For succeeding repeat 
work pieces, the program steps 1-14 of the PROG- 
RAM REPEAT sequence are performed and the 
105 operation is as described hereinbefore with the 
MOVCAM steps operating in accordance with the 
data A, B, C, and D entered for the first repeat work 
piece. 

While there has been illustrated and described 
110 several embodiments of the present invention, it will 
be apparent that various changes and modifications 
thereof will occur to those skilled in the art. For 
example, various types of vision systems are utilized 
in various specific embodiments utilizing various 
115 forms of electromagnetic ware generation by the 
projection unit and complementary sensing devices. 
It is intended in the appended claims to cover all 
such changes and modifications as fall within the 
true spirit and scope of the present invention. 
120 CLAIMS 

1 . Manipulator welding apparatus having a man- 
ipulator arm controllable in one or more axes and 
arranged to perform a weld path on a workpiece at a 
workpiece station, the manipulator welding 

125 apparatus further comprising: 

means which store manipulator arm position data 
during an initial teach phase representing a desired 
taught weld path with respect to a reference work- 
piece at a predetermined location at the workpiece 

130 station; 
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means which project a predetermined light pattern 
on the workpiece at said workpiece station; 

means which store a reference image of said pro- 
jected predetermined light pattern during said teach 
5 phase; 

means which move said manipulator arm along 
said taught weld path in accordance with said stored 
manipulator arm position data during a first repeat 
phase with respect to a second subsequent worl:- 
10 piece; 

means which sense said predetermined projected 
Sight pattern on said second subsequent workpiece 
as said manipulator arm moves along said taught 
weld path during said first repeat phase; 

1 5 image processing means responsive to said sens- 
ing means which generate deviation data represent- 
ing the deviation between the actual weld path of 
said second workpiece and said path described by 
said manipulator arm during said first repeat phase; 

20 means responsive to said generated deviation 
data which generate corrected weld path data rep- 
resenting said actual weld path on said second 
workpiece; and 

means responsive to said corrected weld path data 

25 which move said manipulator arm and which control 
said manipulator welding apparatus to weld said 
second workpiece along said actual weld path dur- 
ing a second repeat phase. 

2. The manipulator welding apparatus of claim 1 
30 further comprising program control means which 

repeat said first and second repeat phases for each 
successive workpiece presented at the work station. 

3. The manipulator welding apparatus of claim 1 
wherein said manipulator arm position data stored 

35 within said initial teach phase represents a plurality 
of points along said taught weld path. 

4. The manipulator welding apparatus of claim 3 
wherein said sensing means is operative to sense an 
image of said predetermined light pattern at each of 

40 said plurality of taught points. 

5. The manipulator welding apparatus of claim 2 
wherein said stored reference image is acquired by 
said sensing means at one or more predetermined 
points along said taught weld path. 

4 5 6. The manipulator welding apparatus of claim 1 
wherein said projecting means is arranged to project 
said predetermined light pattern at a predetermined 
angle with respect to the reference plane of said 
workstation. 

50 7 - The manipulator welding apparatus of claim 1 
wherein said sensing means comprises image detec- 
tion means carried by said manipulator arm. 

8. The manipulator welding apparatus of claim 7 
wherein said manipulator arm comprises a man- 

55 ipulator hand carried at one end of said manipulator 
arm, said manipulator hand comprising a weld gun 
and carrying said image detection means. 

9. The manipulator welding apparatus of claim 8 
wherein said manipulator hand comprises means 

60 which selectively and alternately present either said 
image detection means or said weld gun to said 
workpiece. 

10. The manipulator welding apparatus of claim 
7 wherein said weld gun defines a weld point and 

65 said image detection means in combination with 



said projection means defines an optical center point 
at the intersection of the respective optical axes of 
said projection means and said image detection 
means. 

70 11. The manipulator welding apparatus of claim 
10 wherein said manipulator hand comprises means 
which move said weld gun and said image detection 
means with said weld point and said optical center 
point being coincident. 

75 12. The manipulator welding apparatus of claim 
7 wherein said manipulator hand carries said pro- . 
jecting means. 

13. The manipulator welding apparatus of claim 
1 wherein said manipulator arm comprises man- « 

30 ipulator hand receiving means, said manipulator 
welding apparatus further comprising first man- 
ipulator hand means adapted to interfit with said 
hand receiving means and carrying a weld gun and 
second manipulator hand means adapted to interfit 

S5 with said hand receiving means, said sensing means 
comprising image detection means carried by ssid 
second manipulator hand means. 

14. The manipulator welding apparatus of claim 

13 wherein said second hand means further carries 
90 said projecting means with the optical axes of said 

image detection means and said projecting means 
defining an optical center point. 

15. The manipulator welding apparatus of claim 

14 wherein said projecting means is arranged with 
95 an optical axis forming a predetermined angle with 

respect to the reference plane of the workpiece. 

16. The manipulator welding apparatus of claim 
1 wherein said workpiece includes two or more gen- 
eral planar portions arranged in a predetermine 

100 workpiece configuration and defining said pre- 
determined actual v/eld path. 

17. The manipulator welding apparatus of claim 

16 wherein said projecting means projects a slit pat- 
tern image onto said workpiece. 

105 18. The manipulator welding apparatus of claim 

17 wherein said projected slit pattern image when 
projected onto said workpiece across said actual 
weld path is transformed in accordance with the sur- 
face shape and position of said workpiece. 

110 19- Tne manipulator welding apparatus of claim 

18 wherein said workpiece portions are arranged in 
an overlapping configuration. 

20. The manipulator welding apparatus of claim 
18 wherein said workpiece portions are arranged so r 

115 that said workpiece portions form one or.more pre- 
determined angles with respect to each other. 

21. The manipulator welding apparatus of claim 
18 wherein said workpiece portions are arranged to 
present an actual weld path defined by the respective 

120 abutting edges of said workpiece portions. 

22. The manipulator welding apparatus of claim 
4 wherein said corrected weld path generating 
means is responsive to the actual position of said 
manipulator arm at each of said sensed image 

125 points. 

23. The manipulator welding apparatus of claim 
4 wherein said corrected weld path generating 
means is responsive to said taught point data. 

24. The manipulator welding apparatus of claim 
130 1 further comprising program control means which 



9 



GB 2 087 107 A 9 



store weld condition parameters including weld 
speed. 

25. The manipulator welding apparatus of claim 
1 further comprising program control means which 
5 operate said manipulator arm moving means in said 
first repeat phase at a speed that is at least an order 
of magnitude higher than said speed of said man- 
ipulator arm during said second repeat welding 
movement. 

10 26. Manipulator welding apparatus comprising a 
manipulator arm movable in a plurality of controll- 
able axes and a manipulator hand being carried at 
the end of said manipulator arm, said manipulator 
hand comprising means which carry a weld gun, 

15 means which carry a visual detection system, and 
means which selectively rotate either said weld gun 
or said visual detection system so as to be 
positioned in an operative orientation with respect to 
a reference point on said manipulator arm. 

20 27. Manipulator welding apparatus having a 
manipulator arm controllable in one or more axes 
and arranged to perform a weld path on a workpiece, 
the manipulator welding apparatus further compris- 
ing: 

25 means which store a predetermined projected 
image pattern of a first workpiece at a predeter- 
mined location; 

means which move said manipulator arm along a 
weld path on a second subsequent workpiece, defin- 
30 ing a taught weld path; 

means which sense said predetermined projected 
image pattern along said taught weld path and defin- 
ing the actual weld path of said second workpiece; 
image processing means responsive to said sens- 
35 ing means which determine the deviation between 
said actual weld path and said taught weld path; 

means responsive to said image processing 
means which calculate corrected weld path data rep- 
resenting said actual weld path on said second 
40 workpiece; and 

means responsive to said corrected weld path data 
which weld said second workpiece along said actual 
weld path. 

28. A method for operating manipulator 
45 apparatus having a manipulator arm controllable in 
one or more axes and arranged to perform a welding 
path with respect to a workpiece located at a work- 
piece station, the method comprising the steps of: 
projecting a predetermined light pattern on said 
50 workpiece station; 

storing a reference template image of said pro- 
jected light pattern on a reference workpiece at a 
predetermined location at said workpiece station; 
teaching a desired weld path representing man- 
55 ipulator arm positions along said desired weld path 
on a workpiece; 

sensing said projected light pattern on a work- 
piece as said manipulator arm is moved over said 
taught weld path, said projected light pattern on said 
60 workpiece defining the actual weld path on said 
workpiece; 

calculating the deviation between said actual weld 
path and the path described by said manipulator arm 
during said sensing step in response to said sensing 
65 step; 



calculating corrected weld path data representing 
said actual weld path of said workpiece in accor- 
dance with said calculated deviation; and 

welding said actual weld path on said work piece 
70 by controlling movement of said manipulator arm in 
accordance with said corrected weld path data. 

29. The method of claim 28 wherein said sensing 
step, said deviation calculating step, said corrected 
weld path step, and said welding step are repeated 

75 for each successive workpiece presented to the man- 
ipulator work station. 

30. Manipulator welding apparatus in combina- 
tion with sensing apparatus, image processing 
apparatus, and weld path calculating apparatus sui>- 

80 stantially as described herein with reference to the 
accompanying drawings. 

31 . A method of operating manipulator welding 
apparatus to perform a welding path with respect to 
a workpiece, the method being as substantially 

85 herein described wrth reference to the accompany- 
ing drawings. 
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